Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

• review please @oamg/developers to notify leapp developers of the review request
• /packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp, e.g. from PR#42, use /packit test oamg/leapp#42
Note that first time contributors cannot run tests automatically - they will be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

• /packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
• /packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

/packit build

/packit build

@pirat89 I have tried this patch with a VM that uses a separate virtio block device for /usr and it worked fine. I have verified that /usr is mounted, and that initrd-parse-etc.service was triggered successfully before upgrading (using rd.upgrade.break=leapp-pre-upgrade).

Do you happen to have any recollection on the conditions when this patch was not working?

@MichalHe I have reproduced the problem with:

• fstab

#
# /etc/fstab
# Created by anaconda on Thu May  2 18:40:01 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=fc227809-1070-4d86-b676-54cb81a5455b /boot                   xfs     defaults        0 0
/dev/mapper/rhel01-usr  /usr                    xfs     defaults        0 0
/dev/mapper/rhel00-var  /var                    xfs     defaults        0 0
/dev/mapper/rhel-swap   swap                    swap    defaults        0 0

• partitioning:

# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
# Partition clearing information
clearpart --none --initlabel
# Disk partitioning information
part pv.1349 --fstype="lvmpv" --ondisk=vda --size=7172
part pv.146 --fstype="lvmpv" --ondisk=vda --size=6148
part /boot --fstype="xfs" --ondisk=vda --size=1024
part pv.726 --fstype="lvmpv" --ondisk=vda --size=8196
volgroup rhel00 --pesize=4096 pv.726
volgroup rhel01 --pesize=4096 pv.1349
volgroup rhel --pesize=4096 pv.146
logvol /  --fstype="xfs" --size=3072 --name=root --vgname=rhel
logvol /var  --fstype="xfs" --size=7168 --name=var --vgname=rhel00
logvol /usr  --fstype="xfs" --size=6144 --name=usr --vgname=rhel01
logvol swap  --fstype="swap" --size=2048 --name=swap --vgname=rhel

• lsblk

[root@localhost ~]# lsblk
NAME           MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda            252:0    0  30G  0 disk 
├─vda1         252:1    0   1G  0 part /boot
├─vda2         252:2    0   8G  0 part 
│ └─rhel00-var 253:3    0   7G  0 lvm  /var
├─vda3         252:3    0   7G  0 part 
│ └─rhel01-usr 253:2    0   6G  0 lvm  /usr
├─vda4         252:4    0   1K  0 part 
└─vda5         252:5    0   6G  0 part 
  ├─rhel-root  253:0    0   3G  0 lvm  /
  └─rhel-swap  253:1    0   2G  0 lvm  [SWAP]
vdb            252:16   0  20G  0 disk 

• screenshot with error:
  

note that for the screenshot I had to re-run it, that's why you see an extra msg about .leapp_upgrade_failed file and previous failure.

@pirat89 By default, systemd will mount only / and /usr from /sysroot/etc/fstab. If we want to have any other device mounted early by systemd, we need to add the x-initrd.mount option to corresponding fstab entries. Any such a unit will be picked up and mounted automatically. So, in the reproducer you have shared, I need to modify

/dev/mapper/rhel00-var  /var                    xfs     defaults        0 0

into

/dev/mapper/rhel00-var  /var                    xfs     defaults,x-initrd.mount        0 0

Also, if the device is an LVM's logical volume and the source system is using rd.lvm.lv cmdline args, we need to add anything we will be mounting the the list. So, I needed to add rd.lvm.lv=rhel00/var to the upgrade boot entry in order for the corresponding /dev/mapper/... entry to be present so that the unit generated by systemd could be activated. Alternatively, we can just remove all rd.lvm.lv cmdline args and dracut will make all lvm volumes available.

Otherwise, the upgrade went OK.

Looking at when systemd-fstab-generator is supposed to run: it is supposed to run before any services are run, or when PID 1 is reloaded. Now, initrd does not contain a valid /etc/fstab, so if we run systemd-fstab-generator; systemd daemon-reload it should have no effect whatsoever. Manually creating /etc/fstab inside initrd will not work either - the generated units should mimic fstab, and, thus, they would mount, e.g., /var at /var not /var /sysroot/var, hence they would not help. So we would have to essentially do switchroot, if we want to use systemd-fstab-generator the way normal boot uses it, imho.

What we could do in theory is to run systemd-fstab-generator before reboot and try keeping the generated .mount units. We would then modify them by prefixing mount targets with /sysroot, add them into the upgrade initramfs, and make them our dependencies. This could work, I but have not tried it.

So, the issue is essentially twofold:
1) We stop the standard boot process early, and, therefore, we might be stranded without necessary LVs. For example, if one uses rd.lvm.lv=.. to instruct dracut to only activate volumes for / and /usr, and there is a LV for /var no one will take care of this for us. Luckily, the lvmautoactivation manpage does a great job in explaining how the process works. So, to obtain some magic that will autoactivate LVM for us, we just need:

LEAPP_DRACUT_INSTALL_FILES="/usr/sbin/pvscan /usr/sbin/vgchange /usr/lib/udev/rules.d/69-dm-lvm.rules"

To tell dracut to include key LVM binaries, and an udev rule that takes care of triggering these commands. I have tried booting into the initramfs, and the lv for /var from @pirat89's example is activated, although the rd.lvm.lv= options are used for / and `/usr/

2) We want to ditch running /mount -a in a loop and, instead, use systemd mechanism
We hijack systemd-fstab-generator, keeping the units, but modifying their mount targets to be prefixed with /sysroot. So, I executed systemd-fstab-generator as

# systemd-fstab-generator fstab-mounts fstab-mounts fstab-mounts

resulting in the mount units being placed in /fstab-mounts. Then I deleted units for / (-.mount) and /usr (usr.mount) as these will be already mounted by dracut, leaving me only with var.mount and boot.mount. Then I modified both of these units, prefixing /sysroot to their mount targets. For example, in var.mount we change

Where=/var

into

Where=/sysroot/var

Now systemd is very strict how mount units should be named, so we need to rename the mount units to reflect the new mount target. For example, for /var we have var.mount renamed to sysroot-var.mount and similarly for boot.mount. Finally, systemd-fstab-generator also generates fstab-mounts/local-fs.target.requires directory. The contents of this directory act as if they were listed in Requires= of the local-fs-target. We have changed the names of the mount units, so the symlinks generated by systemd-fstab-generator are broken. Hence:

# cd /fstab-mounts/local-fs.target.requires
# rm ./*
# ln -s ../sysroot-var.mount . 
# ln -s ../sysroot-boot.mount .

Finally, we copy the contents of /fstab-mount to /usr/lib/systemd/system and tell dracut to include these units in our upgrade initramfs:

LEAPP_DRACUT_INSTALL_FILES="$LEAPP_DRACUT_INSTALL_FILES /usr/lib/systemd/system/sysroot-var.mount /usr/lib/systemd/system/local-fs.target.requires/sysroot-var.mount /usr/lib/systemd/system/sysroot-boot.mount /usr/lib/systemd/system/local-fs.target.requires/sysroot-boot.mount"

And we have an upgrade initramfs that a) autoactivates LVM, b) uses systemd to mount entries from /etc/fstab

😁